Skip to content

fix(chat): skip azure async filter events #2255

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Apr 14, 2025

Conversation

kapis
Copy link
Contributor

@kapis kapis commented Mar 24, 2025

  • I understand that this repository is auto-generated and my pull request may not be merged

Changes being requested

When using (Async)ChatCompletionStream, add a weak validation before the accumulation, to ignore events not adhering to the ChatCompletionChunk schema.

This prevents runtime errors when the SSEs are not 100% compatible with OpenAI spec (e.g. when using Azure OpenAI with Asynchronous Filter feature).

Before this fix, the following code will result in an error:

from openai import AsyncAzureOpenAI

client = AsyncAzureOpenAI(
    api_version="2024-08-01-preview",
    azure_endpoint="<azure openai endpoint !!! with Asynchronous Filter enabled !!!>",
    api_key="<api key>",
)

async with client.beta.chat.completions.stream(
    model="deployment-name", 
    messages=[
        {
            "role": "user",
            "content": "hi",
        },
    ],
) as stream:
    async for event in stream:
        print(event)

# ...
# ContentDoneEvent(...)
# then Exception:
File ~/<path>/openai/lib/streaming/chat/_completions.py:397, in ChatCompletionStreamState._accumulate_chunk(self, chunk)
    369 choice_snapshot = completion_snapshot.choices[choice.index]
    370 previous_tool_calls = choice_snapshot.message.tool_calls or []
    372 choice_snapshot.message = cast(
    373     ParsedChatCompletionMessageSnapshot,
    374     construct_type(
    375         type_=ParsedChatCompletionMessageSnapshot,
    376         value=accumulate_delta(
    377             cast(
    378                 "dict[object, object]",
    379                 model_dump(
    380                     choice_snapshot.message,
    381                     # we don't want to serialise / deserialise our custom properties
    382                     # as they won't appear in the delta and we don't want to have to
    383                     # continuosly reparse the content
    384                     exclude=cast(
    385                         # cast required as mypy isn't smart enough to infer `True` here to `Literal[True]`
    386                         IncEx,
    387                         {
    388                             "parsed": True,
    389                             "tool_calls": {
    390                                 idx: {"function": {"parsed_arguments": True}}
    391                                 for idx, _ in enumerate(choice_snapshot.message.tool_calls or [])
    392                             },
    393                         },
    394                     ),
    395                 ),
    396             ),
--> 397             cast("dict[object, object]", choice.delta.to_dict()),
    398         ),
    399     ),
    400 )
    402 # ensure tools that have already been parsed are added back into the newly
    403 # constructed message snapshot
    404 for tool_index, prev_tool in enumerate(previous_tool_calls):

AttributeError: 'NoneType' object has no attribute 'to_dict'

Additional context & links

The primary purpose of this PR is to make this SDK usable with Azure OpenAI Asynchronous Filter feature.

E.g. Langchain internally relies on client.beta.chat.completions.stream in some cases.

An issue describing non-compatible events when using an Asynchronous Feature:

Fixes to the same problem in other popular SDK's:

Other possibly related issues:

CC @kristapratico

@kapis kapis requested a review from a team as a code owner March 24, 2025 21:09
@Fluffet
Copy link

Fluffet commented Apr 1, 2025

I was so happy to see this PR because this has been a real pain. Thank you 🌟

@RobertCraigie RobertCraigie changed the title Fix bug with Azure Async filter events processing fix(chat): skip azure async filter events Apr 14, 2025
@RobertCraigie RobertCraigie changed the base branch from main to next April 14, 2025 09:04
Copy link
Collaborator

@RobertCraigie RobertCraigie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@RobertCraigie RobertCraigie merged commit fd3a38b into openai:next Apr 14, 2025
2 of 3 checks passed
@stainless-app stainless-app bot mentioned this pull request Apr 14, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants